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SYSTEM AND METHOD FOR 
VISUALIZING USER ACTIVITY 

Cross-Reference to Related Application 

This application claims priority to U.S. Provisional Patent Application No. 60/274793 
5 filed on March 9, 200 1 , which is incorporated herein by reference. 

Field of the Invention 

The present invention relates generally to the field of visualizing user activity. More 

-ass? 

*gj specifically, the invention covers the use of visual movies to portray the usage patterns of a user 
O of a web site, software application, or other electronic device. 

JiO Background of the Invention 

As is commonly appreciated in the Internet community, suppliers of information services 
m (for example, web sites, content providers, individuals, including, but not limited to, employees 

□ of an organization, managers, third-parties, executives, and anybody else, hereinafter collectively 

M? 

identified as "Providers") often desire and need to obtain detailed information on how users 
15 utilize the various features and services provided by their sites. Such information is highly 

valuable in determining, for example: glitches in how a web site operates; usage patterns of 

users visiting a web site; what features and functions on a web site capture a user's attention and 

for how long; whether users are directed (either automatically or by choice) to advertisers' sites; 

determining how a user encountered a problem with a particular page on a site; and for other 
20 purposes. In short, Providers commonly desire to know and understand what a user does while 

visiting a site or using an application in order to optimize the visited site for maximum 

effectiveness. 
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Similarly, Providers of software applications (which may or may not be Internet enabled), 
electronic devices (including, for example, microwaves, televisions, VCRs, dishwashers, stoves, 
refrigerators, and other devices), and other devices also desire to be capable of receiving 
information on how a user accesses and utilizes a specific application or device. Such 
5 information is useful for determining, for example, warranty provisions (i.e., a warranty based 
upon a number of wash cycles utilized instead of a number of years), how a user utilizes an 
application or device, and what steps the user performed before they decided that a call to a 
service center was necessary in order to restore the application/device to an original 
configuration instead of its current, undesired configuration, 
fljo Thus, Providers desire to receive information on how a user accesses a web site, 

U application, an electronic device, or other system. For purposes of simplicity only, such 
s information shall collectively be referred to as "Log Data", i.e., data which is suitably logged and 
y identifies user activity. Further, for purposes of simplifying the following discussion of many of 
2 the features and functions of the present invention, web sites, applications, electronic devices, 
Ts and any other device or system for which Log Data may be obtained are herein collectively 
referred in the singular and the plural as a "Web Site." 

Currently, various approaches are utilized to capture Log Data of user activity. One 
common approach utilizes web usability labs in which users are monitored as they utilize a Web 
Site. More specifically, these usability labs are often set up as an array of computers in a lab that 
20 is fitted with two-way mirrors and video recording devices. Sample Web Site users are 

assembled in a focus group and paid to interact with the Web Site being analyzed. Often the 
users are given specific tasks, such as "Purchase product X from the site" or "Locate product 
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information on product Y". The users may also be given random assignments, such as "Go to 

the site and just look around ." 

Further, while the sample Web Site users are working with the Web Site being analyzed, 

all of their actions are being observed by the owners of the site through 2-way mirrors. In 
5 addition, the entire process is preferably recorded on video for follow-up analysis at a later date. 

While this process is often beneficial, it has significant drawbacks. One drawback is that it is 

very expensive, because a Provider has to rent the lab, assemble the test users, record the data 

and later analyze the data before any meaningful benefit may be obtained from the experiment, 
yj Further, this approach is very time consuming as it requires much advance planning and 
S§0 subsequent follow-up. Third, this approach may not result in the identification of the plusses and 
f } minuses inherent in any Web Site. Fourth, this approach does not provide a practical method for 

sharing user activity information throughout a large organization. Lastly, do to the preceding 
%j limitations and others, Providers generally are not able to utilize such an approach with enough 
Q frequency to test changes to a Web Site or other improvements, without repeating the entire 
B6 process. 

Another approach often utilized by Providers in order to assess user activity of a Web 
Site, is to perform statistical analysis of logging information generated during a visit by a user to 
the Web Site. As is common in the art, an entry in a "web log" is generated every time a user 
visits a Web Site or performs a function on the site, for example, the sending of a request and/or 
20 information to a server hosting a web application is recorded on the web log. An example of a 
web log is shown in Figure 1 . More specifically, the web log provides to a Provider an 
indication of: who visited the web site (usually designated by an IP address and an Internet 
Service Provider identifier); when they visited the site (in terms of date and time); for how long 
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they visited a specific page on the site (as designated by the elapsed time between the initial 
downloading of a page and then a subsequent page); and what they visited on the site. In short, 
the web log provides Log Data indicating who, what, where, when, and how information on user 
activity vis-a-vis a web site. Additionally, it is commonly appreciated that similar information 
may also be generated for computer applications and virtually any electronic device (i.e., any 
Web Site) capable of creating user activity files (i.e., user logs). 

Providers desiring information on the utilization of their Web Site often utilize the web 
log to identify statistical representations of how a site is utilized. Such information is commonly 
provided in the forms of charts and tables which are incapable of explaining or illustrating a user 
experience with a Web Site. While this approach eliminates many of the shortcomings of the 
web lab approach (in that it can be timely and is tremendously less expensive), the statistical 
approach also fails to provide Providers with specific illustrations of user usage information of a 
Web Site. 

Therefore, a need exists for an simple, efficient, and effective approach for illustrating to 
Providers how users access a web site. Similarly, manufacturers of electronic devices and 
computer applications (which are not web based) need a simple efficient and effective approach 
for illustrating how a user actually utilizes a device, what function the device was performing 
when an anomalous condition occurred, and similar information. 

Summary of the Invention 

In a first embodiment, the present invention provides a system and method for analyzing 
web server logs (for example, HTTP logs) or other computer generated activity logs and 
converting the information contained on the logs, i.e., the Log Data, into a recreation of a user's 
accessing of a Web Site. The recreation may be provided as a visual, audio, audio/visual, or 
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other presentation (hereinafter, a "Movie") depicting what users were actually 
doing/seeing/hearing when they accessed a Web Site. More specifically, the Movie creates a 
reenactment of what the user was viewing on their screen (or the screen of a device, for example, 
a configuration screen for a television or a VCR connected to a video presentation device), 
5 hearing, or otherwise sensing as the user visited and/or utilized the Web Site. 

The first embodiment also integrates Log Data with other known information about a 
user (if available) and presents that information in the context of the user session. For example, 
e 3 the Log Data may identify a user by an IP address, cookies, or other identifying devices which 
*43 are then matched against a listing of users and IP addresses in order to specifically identify the 
]f|0 user by name, location, and other demographic information. In this manner, the present 
H invention enables Providers to more precisely identify who users are and relate more closely with 
f ss their web "surfing" or other experience. 

Hj In order to create the Movie, especially when a large amount of Log Data is to be 

O processed, the system preferably utilizes two filters: a custom filtering tool for creating a Movie 
1 5 based upon the occurrence of certain events in the Log Data; and an anomaly collector which 

examines key thresholds and then records a user session (i.e., creates a Movie) whenever a target 

event is triggered. 

Additionally, the first embodiment incorporates various reporting and notification 
strategies, including real-time reporting, near real-time reporting, archival reporting, and 
20 statistical reporting. Each Movie is also suitably transferable to others via commonly available 
communications mechanisms including e-mail, server connections, and even U.S. mail delivery 
(for example, a video tape). Thus, the present invention may be utilized by any Provider or 
designee regardless of time and/or location. 
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The first embodiment provides a level of functionality similar to that provided by the web 
labs without the expense, time or system constraints inherent in the web lab approach. The 
removal of such constraints enables the Movie to be produced with a significant saving in time, 
space and money. For example, the present invention is not time dependent, since it may be 
5 accomplished (i.e., a Movie produced) at any time and with any desired frequency. Similarly, 
the Movie can be easily and (if desired) immediately shared with numerous employees in an 
organization including, for example, a Web Site design team, product managers, sales and 
marketing staff, executives, and others. The archiving of Movies in appropriate data storage 
W locations is also provided. 

0 Further, typical applications of the various embodiments of the present invention and the 

y 5 Movie creating capabilities thereof include, but are not limited to, visually displaying the usage 
*' patterns of: a web site user; a user at an Automated Teller Machine (ATM); a user of a 
%S software application (for example, Microsoft Word®); an information kiosk; a car dashboard; 

pi 

Jjf and an appliance (for example, a microwave oven wherein the Movie displays a representation of 

T5 the microwave and a user selecting the various buttons provided thereon). It is to be appreciated 
that the present invention may be suitably configured to present a Movie of any usage of any 
device capable of recording such usage electronically. 

Further, in another embodiment, audio recreations may also be created (i.e., audio 
"Soundtracks"). For example, a recreation of the stations selected by a person while driving in a 

20 car may be provided as a Soundtrack. In such an embodiment, the system records in the Log 
Data not only the stations selected by the user but also the time of day for such selection. The 
Log Data information is then matched with listings of radio broadcasts by date and time to 
recreate a Soundtrack of the user's listening experience. For example, by recalling precise 
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segments of multiple station broadcasts from a recording of the live presentation, and matching 
such portions with the Log Data, a recreation of the actual listening experience for the user may 
be recreated. The Soundtrack would be highly valuable to producers of radio shows, advertisers, 
and "air-time" marketers, because a more precise identification may be obtained of those features 
5 and programs that attracted and retained a listening audience and those features and programs 
which drove the listening audiences to another station. 

The features and functions of the present invention, as discussed with reference to the 
before mentioned embodiments is more fully described and disclosed with reference to the 
following drawing figures, detailed description section, and claims. 

HO Brief Description of the Drawing Figures 

kg Figure 1 is a pictorial illustration of a prior art HTTP log. 

Q Figure 2 is a pictorial representation of one screen of a Movie produced in accordance 

!y with a first embodiment of the present invention. 

P Figure 3 is an illustrative example of a multi-screen presentation depicting two screens in 

15 half screen format specifically selected from four stamp sized screens for a Movie implemented 
according to a first embodiment of the present invention. 

Figure 3 A is an enlargement of the first of the five stamp sized screens presented in 
Figure 3, which has been selected for presentation on a full screen. 

Figure 3B is an enlargement of the second of the five stamp sized screens presented in 
20 Figure 3, which has been selected for presentation on a full screen. 

Figure 3 C is an enlargement of the third of the five stamp sized screens presented in 
Figure 3, which has been selected for presentation on a full screen. 
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Figure 3D is an enlargement of the fourth of the five stamp sized screens presented in 
Figure 3, which has been selected for presentation on a full screen. 

Figure 3E is an enlargement of the fifth of the five stamp sized screens presented in 
Figure 3, which has been selected for presentation on a full screen. 

Figure 4 is an illustrative example of a result showing aggregate traffic distribution as 
calculated by one of the statistical sampling tools provided by the first embodiment of the 
present invention. 

Figure 5 is an illustrative example of an e-mail message delivering a Movie, the e-mail 
being sent by a system implementing the first embodiment of the present invention to a Provider 
subscribing to the system. 

Figure 6 is a exemplary screen shot of an interface provided for a custom filtering tool 
utilized in the first embodiment of the present invention. 

Figure 7 is system flow diagram identifying the system devices and process flow utilized 
in the first, web server based, embodiment of the present invention. 

Detailed Description of the Invention 

As discussed previously, a first embodiment of the present invention provides a system 
and process for generating and communicating to a Provider a Movie depicting how a user 
accessed a Web Site. The Movie is generated based upon Log Data collected by the Web Site, 
and contains information relating to how the user utilized a Web Site. The collecting of Log 
Data for Web Sites is well known in the art, such logging activities are not discussed herein. It is 
to be appreciated, however, that the first embodiment of the present invention may utilize any 
type of Log Data in creating a visual Movie and/or an audio Soundtrack (as appropriate) which 
illustrates a user's activity vis-a-vis a web site, an application, and/or a device (i.e., a "Web 
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Site"). Further, as the multimedia features of the Internet and related applications and devices 
continue to expand it is to be appreciated that user interactivity with a Web Site will include both 
audio and video data (in addition to other forms of data including virtual reality, text, and 
graphics). As such, the first embodiment of the present invention is suitably configured to create 
5 multi-media presentations which encapsulate the user experience, as identified by Log Data files. 
As shown in Figure 2, one illustration of how the first embodiment of the present 
invention provides Movie illustrating a given user's experience with a Web Site, as recorded in a 
f ^ Log Data file, is shown. In this first embodiment, a screen 200 providing a still video image is 
yp presented of the user's experience. In this example, the experience occurred on a web site 
; #0 provided by Amazon.com®. As shown in the view field (202), the Movie portrays a user 
^ shopping and/or considering purchasing a book entitled "Golf for Dummies". More specifically, 
« the view field 202 presents to the Provider the specific screen and/or functions recorded on the 
ftj Log Data file associated with the Amazon server for the user, during the specific session. 
O As shown, the screen 200 preferably contains a border area 204 in which various 

1 5 information relating to the user session is provided. For example, this screen 200 provides 

information relating to an indication of which screen in a series of screens the user accessed 205 
(for example, a user jumping directly to the page 202 would have an indication of page 1 of X 
pages, whereas the user in this example viewed two other screens before reaching the displayed 
screen). Information is also provided identifying the date and time 206 of the user session and 
20 how long 207 the user was on the page. Demographic information about the user is also 

provided as indicated by the IP address for the user 208, and the purchase history for the user 
associated with the IP address 209. The purchase history and other user profiling information 
may be obtained via various well known profiling systems. For example, a cross-reference to a 
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user profile stored in a separate database may be accomplished by utilizing a "cookie" 
communicated by the user's system to the server and suitably recorded in the Log Data. 

Additionally, this screen 200 provides an identification of the referrer 211 which the user 
uses to reach and/or connect to the web site. Additionally, the log data may also be utilized, in 
certain circumstances, (for example, through reverse look-up techniques) to identify the ISP used 
to establish the Internet connection. Further, the referrer, if any, shows how the user found the 
web page displayed in field 202. User feedback information 210 is also provided by tying 
together any e-mails sent to the Provider from the user via the displayed web page. Additionally, 
user feedback information 210 may be pulled from e-mails associated with a customer's record, 
as maintained by the Provider. 

This first embodiment of the present invention also provides a hierarchical traffic 
indication 212 designating how the user accessed the displayed web page. As shown, for this 
example, the user accessed the site from a level immediately below the top level, for example, a 
level probably dedicated to Books. The user then displayed two different web pages relating to 
Books, one of which is shown on the screen 200. 

In addition to displaying the traffic hierarchy in a tree format, this indicator 212 also 
allows the Provider to jump to various other pages in the Movie, without having to rewind, 
restart, or view the Movie in sequential order. Since one embodiment of a Movie may include a 
collection of individual screens which may be presented at a full motion, slow motion, still, or 
other rate, the present invention enables the Provider to view the Movie at any point by suitably 
selecting that frame element associated with a specific segment. For example, by selecting the 
top most level of the highlighted hierarchy 212, the Provider can view the first page accessed by 
the user (i.e., the Movie effectively "rewinds" to the frame associated with that segment of the 
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Movie). Thus, the tree hierarchy 212 enables the Provider to view any element of a user 
experience at any point along the experience. 

Further, this screen 200 also provides sequential progression through the Movie at a 
desired rate, via the forward, backward, and stop buttons 213. Additionally, the screen can be 
saved, e-mailed, and exited by the corresponding buttons 214. 

As shown in Figure 3, the first embodiment of the present invention also allows the 
Provider to view multiple presentations of screens viewed by a user during a session with a Web 
Site. As shown, these multiple screens may be presented in half screen format, "stamp" sized, 
and in various other formats. Additionally, by suitably selecting any one of the "stamps," the 
Provider may see full screen representations of the individual stamps, as shown in Figures 3 A - 
3E. Further, as shown in Figure 3 A, a Movie preferably begins with a screen identifying the 
visitor session by number (for example, when multiple visitor sessions with a Web Site have 
been requested for production into a Movie) and by identifying the site contacted during the 
session (in this example, "ColoradoSkiing.com"). 

Similarly, the first embodiment also creates an ending screen, as shown in Figure 3E, in 
which summary information is presented about the user's experience with the previously 
identified Web Site. More specifically, this ending screen suitably identifies the total time the 
user spent on the site, the number of pages viewed, the reason for leaving the site (when known 
and/or deduced) and any additional information. For example, statistical information pertaining 
to how many users have left the site in the same manner as the current user may be presented. 
Additionally, it is to be appreciated that statistical information and other information may be 
presented on this ending screen or any other screen, as desired and specified by a Provider. The 
presentation of such additional information is preferably controlled by templates and settings 
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designated by a Provider for a specific configuration of this embodiment of the present 
invention. 

As mentioned previously, in addition to providing a Movie and other information relating 
to a user's experience with a Web Site various embodiments of the present invention are capable 
of providing other information related to the user's visit in a graphical or tabular format. An 
example of such information includes, but is not limited to, an aggregation of multiple user visits 
and a median user visit, as shown in Figure 4 (wherein a listing of user visits by site and sub- 
pages is presented). This information is similar to the traffic hierarchy shown in Figure 2, 
however, unlike the Figure 2 presentation (wherein a single user's traffic pattern is displayed), 
the Traffic Distribution chart 400, as shown in Figure 4, provides aggregate information on a 
plurality of user visits. The Traffic Distribution chart may also be suitably presented on any of 
the individual Movie screens (for example, those shown in Figures 3 A-3E), and/or it may be 
presented by itself. Other statistical tabulations and information may also be provided in the 
Movie (during the show and/or at the beginning or end) as desired. Therefore, it is to be 
appreciated that the present invention may utilize other Log Data analysis tools to provide any 
level of detail and/or information relating to the presentation of the Movie and user activity vis- 
a-vis a Web Site. 

Additionally, the first embodiment facilitates the easy and effective distribution of a 
Movie to multiple Providers by attaching the Movie to an e-mail message (as shown in Figure 5) 
which may be forwarded to other Providers as desired. As is well known in the art, various data 
compression and decompression algorithms and processes exist which enable one to 
communicate large quantities of data, as efficiently as possible, via an e-mail or other 
communications medium. The first embodiment of the present invention may utilize any and all 
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such data transmission systems and methodologies in communicating a Movie to single and/or 
multiple users. Additionally, when the communication of a Movie in an e-mail message is 
undesirable (for example, due to system constraints, the size of the Movie file, or other 
variables), the present invention may also be configured to store the Movie on a server or data 
storage device accessible by the Provider (for example, via an Internet or other network 
connection). 

In addition to providing an efficient and proven distribution system for a Movie, the e- 
mail capabilities of the first embodiment also include adding Provider and event specific 
messages to such messages. For example, as shown in Figure 5, the Provider may designate 
specific events, the happening of which on the Provider's Web Site (as captured in the Log Data) 
trigger the creation of a Movie. Instead of randomly providing the Movie, without any 
explanation of context, or requiring an operator to identify and communicate why a particular 
Movie was automatically generated, the first embodiment enables the Provider to establish pre- 
defined notification messages, routed to specific individuals, via various communications 
mediums (including, for example, e-mail, page, and telephone). Thus, the first embodiment of 
the present invention enables Providers to specify how, when, and in what format a notification 
message and/or a Movie will be provided by the system to the Provider and others. 

Figure 6 provides one illustration of an application interface screen for a Filtering 
Administration Tool (as explained in greater detail below) used in the system of the first 
embodiment. This screen provides an interface through which a Provider specifies when Movies 
are to be generated, how played back, and when they are to be notified of the creation of such 
Movie. As shown, this screen 600 includes a field 602 in which the Provider may specify 
filtering conditions (the filter function of the present invention is discussed in greater detail 



13 



Attorney Docket No. 5266.02 

below). Such filtering conditions include, but are not limited to, filtering based upon the 
referring site (i.e., from which site the user is contacting the web site of concern), by common 
actions (i.e., does the user purchase an item, provide feedback, not buy an item, or abandon a 
previously loaded shopping cart), by timing (i.e., the time of day and/or the day of the week), and 
5 by other variables (i.e., is the user a first time visitor, a repeat visitor, the most popular path 
utilized, and other variables). By appropriately specifying the filtering conditions the Provider 
may tailor the system such that the system reviews the Log Data and creates a Movie related to 
those aspects of a user's experience at a web site that the Provider is most concerned about. For 
JI example, a marketing person might request the system to generate a Movie based upon a given 
AO set of Log Data, such as the Most Popular Path utilized by an aggregate user. Such a Movie may 
N= enable the marketing person to identify those web pages a visitor is most likely to click through 
^ in reaching a destination and thereby design marketing and advertising features geared to such 
J; path (for example, pop-up windows that automatically appear advertising a product upon the 
PI selection of a web page along the most popular path). 

15 This screen 600 also includes a field 604 in which a Provider may specify how a Movie is 

played back to them or others. More specifically, this feature enables the Provider to specify that 
the Movie will automatically be generated on a given display device (for example, a video 
monitor of a control center). This automatic viewing function might be very popular, for 
example, with system administrators who desire to receive an automatic movie whenever a 

20 hacker gains access to a secure site, thereby enabling the administrator to determine and close the 
portal through which the hacker gained access. The field 604 also enables the Provider to 
playback a Movie at varying speeds including, but not limited to, the real time rate for the visitor, 
on a manual advance setting, or on various accelerated rates. Other control features regarding 
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the playback and presentation of the Movie may be added/deleted to this screen and others as 
desired. 

As mentioned earlier herein, the first embodiment enables the Provider to specify how, 
when and where they are to be notified about a Movie. When a Movie is automatically 
5 generated, this notification control function may be extremely valuable because it enables 

Providers to continue with their day-to-day activities without concern as to whether the present 
invention is appropriately monitoring and screening the Log Data for events of interest and 
creating a corresponding Movie thereof. Further, in this first embodiment, the Provider may also 
^ specify the system to Save the Movie and/or E-Mail the Movie automatically or manually. Other 
gO functions may also be provided, as desired, with appropriate modifications to the interfaces, 
pi system and process flows. 

H Referring now to Figure 7, an embodiment of a system and a process by which the 

O present invention provides a Movie to a Provider is illustrated. As shown the system is broken 

aw ; 

lg out into three segments. The first segment includes those User System segment components 
j|J5 which are commonly used to generate Log Data. Also shown are Server Side system 

components, which are utilized to generate a Movie based upon Log Data. Lastly, Provider Side 
system components, which are commonly utilized by a Provider to receive and review a Movie, 
are depicted. Each of the components in these systems are described in greater detail herein 
below. 

20 As shown and discussed previously, the system preferably utilizes web usage Log Data 

706 to generate the Movie visualizing the user activity vis-a-vis the Web Site. In an Internet 
application of the present invention entries into the Log Data are preferably made every time a 
user device 700 is connected to a web server 704 via the Internet 702 or another direct/indirect 
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connection. The collection of Log Data for such user connections is well known in the art. 
Additionally, it is to be appreciated that this embodiment may also be adapted to non-Internet 
applications, including one for applications and/or devices which are configured to record and/or 
utilize Log Data 706 collected for a Web Site (i.e., a web site, an application or an electronic 
device 701 (for example, a television)). 

Regardless of how, when or where the Log Data 706 is gathered, compiled and/or stored, 
this embodiment suitably accesses such Log Data 706 and analyzes the data in the Log Analyzer 
708. The Log Analyzer 708 is, in its most fundamental form, a user configurable analyzer that 
receives an inputted data stream and outputs a product of the analysis of the inputted data stream. 
In this case, the inputted data stream is generally a stream of Log Data provided, for example, by 
a web server. The Log Analyzer 708 may receive the stream of Log Data directly from a web 
server, from an intermediary (for example, a data storage device), or on a real-time basis, for 
example, when requests and information from a user are being provided to the web server a copy 
of such request, information, and the pages returned by the web server may simultaneously, or 
soon thereafter, be provided to the Log Analyzer. Additionally, the Log Analyzer 708 may be 
configured to access the Log Data 706 on a scheduled, intermittent, periodic, and/or as needed 
basis. It is to be appreciated that based upon how often, how much, how long, and the type of 
Log Data 706 analysis required for a given application, the Log Analyzer 708 may include or 
have access to numerous data storage devices, as needed, in order to process such Log Data. 

Further, as shown in Figure 7, the Log Analyzer 708 is in communication with a Filter 
710, an Anomaly Collector 712, a Notifier 720, and a Video Reporter 716. The function of each 
of these components are described in greater detail below. 



16 



Attorney Docket No. 5266.02 

More specifically, the Log Analyzer 708 obtains analysis requests from either the Filter 
710 and/or the Anomaly Collector 712. These requests direct the Log Analyzer 708 as to which 
Log Data 706 to obtain and for which patterns, events, or any other items of information the Log 
Analyzer is to search while the Log Analyzer 708 is reviewing the Log Data 706. As such, the 
Log Analyzer 708 performs those searching and filtering functions directed to it by the Filter 710 
and/or the Anomaly Collector 712. 

Additionally, when a request is received (from either the Filter 710 or the Anomaly 
Collector 712), the Log Analyzer 708 suitably obtains a relevant portion of the listings of a 
user's activities from the Log Data 706. It is to be appreciated that this listing may be as large or 
as small as desired. For example, a Provider (for example, a law enforcement officer) requesting 
a search for Log Data showing a user performing a specific illegal function might entail 
searching numerous days or even months worth of data. Similarly, a Provider requesting a 
search for users selecting a specific banner advertisement on a specific web site might utilize a 
Log Data file that is comparably small Thus, the Log Analyzer 708 may analyze any quantity of 
information and based upon the results of such analysis, provide an output indicative thereof. 

It is also to be appreciated that the output from the Log Analyzer 708 may significantly 
vary from request to request. However, in the first embodiment, the Log Analyzer 708 typically 
provides an output indicating a sequence of pages viewed (with unique URLs and date/time 
stamps for each page), a referring site, a browser type used, and any unique filters for the user 
visiting the site or application. These sequences of pages are preferably the data utilized by the 
Video Reporter 716 to create still frames that illustrate, in sequential order, those pages a user 
accessed while connected to a specific Web Site that generates the Log Data 706. 
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Additionally, in another embodiment, the output from the Log Analyzer 708 may also 
include "Fill-Ins". Fill-Ins are information that is provided by the Log Analyzer 708 that is not 
physically present in the Log Data 706 but whose existence may be identified based upon an 
extrapolation of the first page and the subsequent pages a user viewed (or in the case of an 
electronic device, the first button pushed and a subsequent button pushed where, for example, an 
intermediate step was not recorded). One example of a Fill-In is a movement of a cursor across a 
web page to select a banner advertisement. Those skilled in the art appreciate that Log Data 
generally does not include cursor movement information. However, based upon the fact that a 
user was viewing page "x" and then a request to view a page associated with a banner 
advertisement was received by the web server and recorded the Log Data, the Log Analyzer can 
extrapolate the user's actions prior to selecting the banner advertisement (i.e., the user 
repositioned their cursor over the banner advertisement). Similarly, data provided in data entry 
fields (for example, a name and credit card number) may also be simulated in the Movie 
presented to the Provider using the Fill-In techniques of the third embodiment of the present 
invention. 

As mentioned previously, the Log Analyzer 708 is preferably connected to both a Filter 
710 and an Anomaly Collector 712. The Filter 710 provides an interface between the Log 
Analyzer 708 (i.e., the component actually accomplishing the filtering of the Log Data 706) and 
a Filtering Administration Tool 734. As discussed previously, an interface screen for the various 
embodiments of a Filtering Administration Tool is shown in Figure 6 as the Search Tool More 
specifically, the Filter 710 formats requests received from the Filtering Administration Tool 734 
into a format desired by the Log Analyzer 708. Such requests preferably contain the name of the 
Provider that created the filter request, the criteria upon which to conduct the searching of the 
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Log Data 706, when to run the search (i.e., when to filter the Log Data), and at what time periods 
the Log Analyzer 708 is to analyze, if at all, the resulting information provided from the filtering 
of the Log Data. It is to be appreciated that the time at which the filtering is conducted may be 
different than the time at which the analysis is conducted. For example, a filter request may 
direct the Log Analyzer to identify every user session wherein the user clicked on a banner 
advertisement. The same request may also direct the Log Analyzer to save the results of such 
filtering and at an appropriate time (for example, after 10 samples have been obtained) to analyze 
the filtered data using statistical analyses and other techniques. Additionally, the Tool 734 
provides fields in which the Provider specifies how they are to be contacted when an anomaly 
occurs. 

The Filter Administration Tool 734 provides an interface between the Provider and the 
Filter 710. More specifically, this Tool 734 provides templates, interface screens and other 
devices which enable the Provider to specify criteria for filtering requests of Log Data. This 
Tool 734 also allows a Provider to save requests and share requests with others as desired. 

Referring still to Figure 7, the Log Analyzer 708 is also connected to the Anomaly 
Collector 712. The Anomaly Collector 712 is similar in many aspects to the Filter 710 in that it 
directs the Log Analyzer 708 to analyze the Log Data 706 based upon specific criteria. 
However, unlike the Filter 710, the Anomaly Collector does not direct the Log Analyzer to 
search for instances of specific user activity. Instead, it directs the Log Analyzer to identify 
anomalous events (for example, when a user is suddenly booted from a web server due to a glitch 
in the system). In the first embodiment, the Anomaly Collector 712 directs the Log Analyzer 
708 to search for anomalies every 30 minutes, however, other time periods may be utilized as 
desired. 
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When an anomaly occurs, the Anomaly Collector 712 is provided with an indication of 
the anomaly by the Log Analyzer 708. The Anomaly Collector 712 provides a message to the 
Notifier 720. This message preferably contains the name of the anomaly, the Provider account 
that needs to be notified, and any other information, as specified by the Provider via the 
5 Anomaly Collector Administration Tool 736. 

The Anomaly Collector Administration Tool 736 provides an interface between the 
Provider and the Anomaly Collector 712. More specifically, this Tool 736 is similar to the Filter 
Admin Tool 734 in that it provides an application (e.g., software) and/or a device (e.g., 
lfl hardware) upon which an interface may be presented to the Provider for specifying criteria used 
Q|0 to detect an anomaly and the reporting thereof When configured as an application, the Anomaly 
Collector Admin Tool 736 and the Filter Admin Tool 734 are generally hosted, for example, on a 
^ web browser on a compatible device (for example, a computer workstation, laptop computer, 
y personal data assistant, web enabled wireless communications device, and other devices). The 
0 Tools 734 and 736 are preferably forms which capture user requests (for filtering and anomaly 
1*5 collecting, respectively). More specifically, the form for the Anomaly Collector Admin Tool 
736 utilizes fields in which a Provider may specify a date (or time period) during which to 
perform the anomaly collecting and analysis, trigger events (for example, the number of times a 
banner advertisement is clicked or the number of users exiting a certain page), and threshold 
levels for trigger events. Additionally, the Tool 734 provides fields in which the Provider 
20 specifies how they are to be contacted when an anomaly event occurs. 

Another component connected, either directly or indirectly, to the Log Analyzer 708 is 
the Video Reporter 716. The Video Reporter 716 is the component which receives those 
segments of the Log Data 706 identified by the Log Analyzer 708 as meeting a given set of 
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filtering criteria and/or anomaly criteria. Upon receipt of the segments, the Video Reporter 716 
converts such segments into a Movie. More specifically, as the Log Analyzer 708 identifies 
segments of the Log Data 706 (wherein a segment may be a single entry in the Log Data, a 
sequence of entries, a collection of entries, or an entire batch of entries), the segments are 
5 provided either directly or indirectly (i.e., through the Filter 710 and/or the Anomaly Collector 
712) to the Video Reporter 716. The Video Reporter 716 utilizes the received segments to create 
the Movie. 

As discussed above, the Movie may be presented in numerous formats, including, but not 
limited to, as a still image, as a sequence of still images, as a series of moving images, as 



Q0 statistical data with or without images, as a web page, and in other formats. Similarly, the Movie 
ffl may be presented via any communications medium and/or signal format including digital, 
"H analog, satellite, cable, Internet, wireless mediums, radio frequency mediums, wired mediums, 
j^j compressed, uncompressed, broadcast, MPEG format, DVD format, and any other mediums 
J and/or formats. Further, any single enactment of a Movie may often be a unique presentation of 
jJS images, graphics, text, audio, virtual reality, and/or other data formats. In order to create this 
presentation (i.e., the Movie), the Video Reporter suitably obtains a copy of any web pages or 
other information sites identified in the Segment. In order to simulate the condition of the web 
page as it was presented to the user, the Video Reporter suitably provides any data variables or 
other inputs necessary to configure the web page as a duplicate of that provided to and originally 
20 presented to the user. However, appropriate security protections are provided in the Video 
Reporter so that confidential information (for example, a user's credit card number) may be 
masked as necessary. 
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Further, upon obtaining any web pages or other information, the Video Reporter 716 
formats the data such that it is compatible with a viewing device specified by the Provider. A 
Video Reporter Admin Tool 738 preferably provides an interface between the Provider and the 
Video Reporter. As provided for the other Tools 734 and 73 6, the Video Reporter Admin Tool 
738 provides those templates upon which the Provider may specify the format (and other 
variables) in which a Movie is to be presented. Additionally, this Tool 738 includes a search 
interface which enables the Provider to specify specific Movies for presentation, such Movies 
being saved in and retrieved from a Data storage device 718 automatically and/or upon a 
Provider's instruction. 

For example, a Provider may request a search for "all movies involving visitors to 
Amazon.com" or to "show all movies from January 17, 2000 between 8:00 am and 9:00 am 
MST." The Tool 738 initiates a search by the Video Reporter 716 of the appropriate Data 
storage devices 718 or other archives for all relevant Movies. Such searching is accomplished 
via standard database queries. Further, the Movies and indexes thereof may be suitably stored in 
a relational database, object based database, or other database structure. The storage and 
retrieval of information in/from all types of databases is well known in the art. 

The first embodiment also provides a Notifier 720. The Notifier performs the function of 
sending notification messages to the designated Provider(s). Preferably, such notification 
messages are sent via e-mail 722, however, other messaging formats and protocols including 
faxes 724 and other communications mediums 726 (for example, telephones, pagers, personal 
data assistants and other wireless or wired devices) may be utilized. A typical message (as 
shown above in Figure 5 for an e-mail message) contains the following information: the name of 
the recipient of the notification message, the date and time of the event, the name of the event, 
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and a hyperlink (when possible) to direct the Provider to the data file containing the Movie. 
More or less information may be provided in a message as particular needs and communications 
mediums dictate. As discussed previously, the Notifier 720 may also be configured to e-mail a 
notification to multiple individuals, as desired. Additionally, when a message is generated, such 
message may be forward to other users as desired and as is common in the art. 

The final component of the system is the device upon which the client views the Movie. 
Such devices are represented in Figure 7 by the Web Browser 728, the Movie Viewer 730 
(which may stream media), and the Other Viewers 732. When the Provider configures the Video 
Reporter 716, via the Video Reporter Admin Tool 738, a designation is made of the type(s) of 
viewer upon which the Provider desires to view a given Movie. It is to be appreciated that a 
Provider may request a Movie to be presented in any format and/or number of formats. As such, 
the present invention is Provider device neutral as the configuration of the Movie is determined 
by the Video Reporter 716 prior to transmission to the Provider. In this regard, any device(s) 
capable of presenting a given type of data may be utilized by the Provider. Similarly, multiple 
instances of a Movie may be provided (simultaneously or separately) to various types of viewing 
devices. 

For example, in an electronic device (or appliance) embodiment, a Provider might be a 
service technician who suitably downloads from the electronic device in question a Log Data file 
which has recorded a given amount of time prior to a given event (for example, the 
malfunctioning of the electronic device). This information, after being appropriately filtered 
and/or analyzed, may be presented in the format of a Movie. Additionally, the Movie may be 
configured for presentation on a type of device specified by the Provider (for example, a 
computer lap top or a personal data assistant). Additionally, the system may be directed to 
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present such information in a format useful to the Provider. For example, a service technician 
may desire to see various scenes depicting the operating state of the electronic device in question 
prior to, at the time of, and after the malfunction. Such scenes in the Movie may include, for 
example, a clocking diagram of a bus controller, a series of user actions (for example, a series of 
5 buttons pressed by the user), and/or an indication of any alarm signals or messages generated by 
the electronic device. However, in order to present such scenes in a Movie, it is to be 
appreciated that the electronic device must create a Log Data file with sufficient information to 
q recreate such information directly or by using a Fill-in. 

tfj Additionally, the various embodiments of the present invention enable the Movie to be 

|j 0 presented to the Provider in at least two formats (other formats, for example, an audio format, 
^ may also be provided). The two preferred formats are as either a web page presented on a Web 
m Browser 728 or as a data file presented on an application specific Other Viewer 732 (for 
py example, one similar to a Napster music viewer that does not run directly on a web browser and 
0 is instead a stand alone application). Other formats may also be utilized as such formats become 
1 5 commonly available. When the Web Browser approach is utilized, the Movie is presented 

directly in the browser via common approaches such as a GIF image on the Web Browser 728, or 
a video stream in a web browser based application such as a RealVideo® Player, Windows 
Media Player®, or Apple QuickTime® on the Movie Viewer 730. 

When the Movie is presented via the Other Viewer 732, a stand alone application is 
20 deployed to all Providers. The stand alone application preferably connects to other components 
of the system using standard Internet protocols (e.g., TCP/IP) without requiring the use of a 
browser. As is commonly appreciated, such an application can be written in various languages 
including, but not limited to, C++, VisualBasic, and Java. 
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Regardless of the specific embodiment of client viewer utilized, the viewer preferably 
allows the Provider to control the speed of playback, and other parameters (as discussed above in 
reference to Figure 5). The Movie files may also be annotated, cut, pasted, copied, and similarly 
manipulated using commonly available functions. 

As such, the before mentioned embodiments of the present invention provides a system 
and process for creating a visual or other representation of user activity. As discussed above, the 
present invention utilizes various components, some of which are commonly available. 
However, it is to be appreciated that the foregoing embodiments of the present invention are not 
limited to any specific networks, communications mediums, platforms (including, for example, 
Windows, Unix, and Macintosh), or protocols. Further, the various embodiments of the present 
invention may utilize any communication paths, file types, applications, components, devices, 
protocols, and the like to provide the features and functions identified herein and set forth in the 
claims. Thus, the various embodiments of the present invention may be utilized to "visualize" 
(which is herein defined to include any human perceptible signal including, but not limited to 
audio, video, and tactile signals) or user activity for any Web Site regardless of the 
communications paths utilized by the user, the format of the data accessed, the presentation 
device utilized to present the visualization, and any other variable or factor to create and present 
a Movie. 
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